Skip to content

Move recalculate off the main thread and add durable backfill#207

Merged
coji merged 9 commits intomainfrom
improve/worker-thread-analyze
Mar 17, 2026
Merged

Move recalculate off the main thread and add durable backfill#207
coji merged 9 commits intomainfrom
improve/worker-thread-analyze

Conversation

@coji
Copy link
Owner

@coji coji commented Mar 17, 2026

Summary

  • run recalculate analyze/upsert work in worker threads so synchronous SQLite work does not block the main thread
  • add SQLite contention handling with busy_timeout, worker retries, step timing logs, and sqlite-busy summaries
  • add a durably-backed backfill job and switch the CLI command to trigger it through durably

Validation

  • ran node --import tsx batch/cli.ts recalculate aScaf_RvvFiOJWD3m8X32
  • ran DATABASE_URL="file:/Users/coji/progs/upflow/data/data.db" NODE_ENV=production node --import tsx batch/cli.ts recalculate aScaf_RvvFiOJWD3m8X32
  • ran node --import tsx batch/cli.ts backfill aScaf_RvvFiOJWD3m8X32 --files
  • ran node --import tsx batch/cli.ts recalculate aScaf_RvvFiOJWD3m8X32 after backfill completed
  • pnpm validate passed locally

Summary by CodeRabbit

リリースノート

  • New Features

    • バックフィルを耐久ジョブとして実行・管理できるようになり、ジョブ起動時に進捗と結果が可視化されます。
  • Chores

    • バックフィルコマンドをジョブ駆動に変更、進捗・ログ統合とヘルプ文言を更新しました。
  • Bug Fixes / Reliability

    • SQLiteのビジーイベントを集計してログ出力するようにし、診断性と安定性を向上しました。
    • 一部データ処理のバッチ/実行挙動を調整して堅牢性を改善しました。
  • Documentation

    • AIエージェント向けガイド(AGENTS.md)を追加しました。

@coderabbitai
Copy link

coderabbitai bot commented Mar 17, 2026

Warning

Rate limit exceeded

@coji has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 22 minutes and 20 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7c5fb822-1bda-47ad-884e-6ca4af6e7950

📥 Commits

Reviewing files that changed from the base of the PR and between 53ff5a5 and d28d826.

📒 Files selected for processing (1)
  • app/services/jobs/backfill.server.ts
📝 Walkthrough

Walkthrough

Durably に backfill ジョブを追加し、CLI から Durably 経由でバッチ実行するよう切替。upsert ワーカー削除と runUpsertInWorker の撤去、SQLite ビジーイベント収集の導入、WAL チェックポイント挙動と WAL ジャーナルサイズ制限の調整、バッチ処理のチャンク/イールド改善を実装。

Changes

Cohort / File(s) Summary
Durably ジョブ登録
app/services/durably.server.ts
backfillJob をインポートして Durably のジョブ設定に登録。
バックフィルジョブ実装
app/services/jobs/backfill.server.ts
新規 backfillJob を追加。organizationIdfiles を受け、各リポジトリを step.run で処理して処理数を返す。トークン検証とステップ毎の進捗分離を備える。
CLI / バックフィルコマンド
batch/cli.ts, batch/commands/backfill.ts
CLI 表示文を更新。backfill コマンドを Durably の durably.jobs.backfill.triggerAndWait 呼出しへ変更し、進捗・ログマッピングと durably.stop() クリーンアップを追加。
ワーカー / 解析共通処理
app/services/jobs/shared-steps.server.ts, app/services/jobs/run-in-worker.ts, app/services/jobs/run-in-worker.test.ts
SQLite ビジーイベント収集・要約を導入。runAnalyzeInWorker 呼出しに onSqliteBusy コールバックを渡すよう変更。runUpsertInWorker と Upsert 型/エントリポイントを削除し、WorkerEntrypoint を analyze-worker に限定。テスト期待値を対応変更。
ワーカー実装削除
app/services/jobs/upsert-worker.ts, package.json (build:workers)
upsert-worker.ts を削除し、workers ビルド設定から upsert-worker を除外。
SQLite/Journaling 調整
app/services/tenant-db.server.ts, app/services/jobs/crawl.server.ts
テナント DB に journal_size_limit を追加(64MB)。一部 finalize から WAL チェックポイント呼び出しを削除。
DB バッチ処理の改善
batch/db/mutations.ts
chunk サイズ既定値を縮小し、各チャンク後に event loop に戻る yieldToEventLoop を挿入。batchReplacePullRequestReviewers をチャンク毎にトランザクション化。
ドキュメント / .gitignore
.gitignore, AGENTS.md
.gitignore から AGENTS.md を除外(追跡対象に)。新規 AGENTS.md を追加(AI エージェント運用ガイド)。

Sequence Diagram(s)

sequenceDiagram
    participant CLI as CLI/Command
    participant Durably as Durably Instance
    participant Job as Backfill Job
    participant Org as Organization Service
    participant Repo as Repository Handler
    participant Worker as Worker Process

    CLI->>Durably: triggerAndWait(organizationId, files)
    Durably->>Job: run(step, input)
    Job->>Org: getOrganization(organizationId)
    Org-->>Job: organization data
    Job->>Job: validate token
    loop each repository
        Job->>Repo: step.run(backfillRepo) with progress
        Repo->>Worker: analyze(..., onSqliteBusy)
        Worker-->>Repo: analyze result + busy events
        Repo->>Worker: upsert(...)  %% upsert runs via DB path after refactor
        Worker-->>Repo: upsert result
        Repo-->>Job: repository processed
        Job->>Job: update progress
    end
    Job-->>Durably: return { repositoryCount }
    Durably-->>CLI: success with repositoryCount
    CLI->>Durably: stop()
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Poem

🐰 新しいジョブで跳ぶよ、ぴょんぴょん
リポごとに数えて、進捗が増えるよ
ビジーな音も記録して、あとでまとめる
小さな調整で道は軽くなる、ぴょん✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed タイトルはプルリクエストの主な変更を正確に反映しており、「recalculateをメインスレッドから移動」と「耐久性のあるbackfillを追加」という2つの重要な目的を簡潔に述べています。

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch improve/worker-thread-analyze
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
app/services/jobs/shared-steps.server.ts (1)

196-199: busy イベントのログ出力レベルについて確認してください。

SQLite busy イベントが発生した場合、step.log.warn で警告として出力されています。これはオペレーターへの通知として適切ですが、gaveUp イベントが含まれる場合は、より深刻な問題を示している可能性があります。

gaveUp が 0 より大きい場合は error レベルでログ出力することを検討してください。

♻️ gaveUp がある場合は error レベルで出力する提案
   const busySummary = summarizeBusyEvents(sqliteBusyEvents)
   if (busySummary) {
-    step.log.warn(busySummary)
+    const hasGaveUp = sqliteBusyEvents.some((e) => e.gaveUp)
+    if (hasGaveUp) {
+      step.log.error(busySummary)
+    } else {
+      step.log.warn(busySummary)
+    }
   }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@app/services/jobs/shared-steps.server.ts` around lines 196 - 199, The current
code logs the summarized busy events with step.log.warn unconditionally; update
the handling after calling summarizeBusyEvents(sqliteBusyEvents) so that if the
returned busySummary indicates gaveUp > 0 (inspect busySummary.gaveUp) you call
step.log.error with the busySummary, otherwise keep step.log.warn; ensure you
still only log when busySummary is truthy and reference the
summarizeBusyEvents/sqliteBusyEvents variables and the step.log.warn/error calls
when making the change.
app/services/jobs/backfill.server.ts (1)

37-42: リポジトリ単位のエラーハンドリングを検討してください。

現在の実装では、1つのリポジトリで backfillRepo が失敗すると、ジョブ全体が停止します。他のリポジトリの処理を継続し、失敗したリポジトリのみを後で再試行できるようにする方が、大規模な組織では有用かもしれません。

Durably がステップ失敗時にリトライするため、現在の設計でも問題ありませんが、部分的な成功を許容するかどうかは要件次第です。

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@app/services/jobs/backfill.server.ts` around lines 37 - 42, Wrap the per-repo
call to backfillRepo inside the step.run callback with a try/catch so a failure
for one repository doesn't bubble out and stop the whole job: inside the loop
that calls step.run(`backfill:${repoLabel}`...) catch any error from
backfillRepo(repository, ...) and log it with the repoLabel and error details,
push the repository identifier into a failedRepos array declared outside the
loop for later retries, and do not rethrow so the loop continues; optionally,
after the loop finishes, check failedRepos and decide whether to surface a
job-level error or schedule retries.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@app/services/jobs/backfill.server.ts`:
- Around line 37-42: Wrap the per-repo call to backfillRepo inside the step.run
callback with a try/catch so a failure for one repository doesn't bubble out and
stop the whole job: inside the loop that calls
step.run(`backfill:${repoLabel}`...) catch any error from
backfillRepo(repository, ...) and log it with the repoLabel and error details,
push the repository identifier into a failedRepos array declared outside the
loop for later retries, and do not rethrow so the loop continues; optionally,
after the loop finishes, check failedRepos and decide whether to surface a
job-level error or schedule retries.

In `@app/services/jobs/shared-steps.server.ts`:
- Around line 196-199: The current code logs the summarized busy events with
step.log.warn unconditionally; update the handling after calling
summarizeBusyEvents(sqliteBusyEvents) so that if the returned busySummary
indicates gaveUp > 0 (inspect busySummary.gaveUp) you call step.log.error with
the busySummary, otherwise keep step.log.warn; ensure you still only log when
busySummary is truthy and reference the summarizeBusyEvents/sqliteBusyEvents
variables and the step.log.warn/error calls when making the change.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4997b8fd-425d-4eb1-975f-cc86a03125f0

📥 Commits

Reviewing files that changed from the base of the PR and between ce1ed2f and 3194dd2.

📒 Files selected for processing (5)
  • app/services/durably.server.ts
  • app/services/jobs/backfill.server.ts
  • app/services/jobs/shared-steps.server.ts
  • batch/cli.ts
  • batch/commands/backfill.ts

@coji coji merged commit eb77957 into main Mar 17, 2026
6 checks passed
@coji coji deleted the improve/worker-thread-analyze branch March 17, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant